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(57) Abstract 

A method for tracking motion from field to field in a sequence of related video broadcast images. The method uses template correlation 
to follow a set of predetermined landmarks (26) within a scene in order to provide position information of objects in the current image 
(20). The current image object position information is compared to position information of the same objects within a reference array data 
table. The comparison is accomplished through the use of warp equations that map points in the cument image to points in the reference 
array. Motion is tracked according to a velocity prediction scheme utilizing a weighted formula that emphasizes the weight of landmarks 
that are closer to their predicted position. 
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Title 

METHOD OF TRACKING SCENE MOTION 
FOR LIVE VIDEO INSERTION SYSTEMS 



Cross-Referen ce to Related Applications 

This patent application is related to, and claims the priority of U.S. Provisional Patent 
Application No. 60/000,279 filed on June 16, 1995 entitled "APPARATUS AND METHOD OF 
INSERTING IMAGES INTO TELEVISION DISPLAYS" and is related to U.S. Patent 
Application filed November 28, 1995 entitled "SYSTEM AND METHOD FOR INSERTING 
STATIC AND DYNAMIC IMAGES INTO A LIVE VIDEO BROADCAST" and U.S. Patent 
Application No. 08/381,088 filed January 31, 1995 entitled "LIVE VIDEO INSERTION 
SYSTEM". 



Background of the Invention 

1. Field of the Invention 

This invention relates to the improved performance of devices for inserting realistic indicia 
into video images. 

2. Description of the Related Art. 

Electronic devices for inserting electronic images into live video signals, such as described 
in U.S. Patent 5,264,933 by Rosser, et al., have been developed and used for the purpose of 
inserting advertising and other indicia into broadcast events, primarily sports events. These 
devices are capable of seamlessly and realistically incorporating logos or other indicia into the 
original video in real time, even as the original scene is zoomed, panned, or otherwise altered in 
size or perspective. 

Making the inserted indicia look as if it is actually in the scene is an important but difficult 
aspect of implementing the technology. A troublesome aspect is that the eye of the average viewer 
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is very sensitive to small changes in the relative position of objects from field to field. 
Experimentally, instances have been found where relative motion of an inserted logo by as little 
as one tenth of one pixel of an NTSC television image is perceptible to a viewer. Placing, and 
consistently maintaining to a high precision, an inserted indicia in a broadcast environment is 
crucial in making video insertion technology commercially viable. A broadcast environment 
includes image noise, the presence of sudden rapid camera motion, the sporadic occurrence of 
moving objects which may obscure a considerable fraction of the image, distortions in the image 
due to lens characteristics and changing light levels, induced either by natural conditions or by 
operator adjustment, and the vertical interlacing of television signals. 

In the prior art, the automatic tracking of image motion has generally been performed by 
two different methods. The first method, described in U.S. Patent No. 4,084,184 issued to D.W. 
Crain, uses sensors placed on the camera to provide focal distance, bearing and elevation 
information. The second method examined the image itself and either followed known landmarks 
in the video scene, using correlation or difference techniques, or calculated motion using well 
known techniques of optical flow. See, Horn, B.K.P. and Schunck, B.G., '•Determining Optical 
Flow", Artificial Intelligence, pp 185-203 (1981). Landmarks may be transient or permanent and 
may be a natural part of the scene or introduced artificially. A change in shape and pose of the 
landmarks is measured and used to insert the required indicia. 

The problem with the first method is twofold. First, it requires special cameras at the 
event thus limiting the situations in which live video insertion can be undertaken. Second, the 
desired precision can only be achieved with expensive equipment attached to each camera. As 
an example, a 1 in 7000 placement precision in a typical baseball scene covering about 7 meters 
of play and shot from 300 meters away requires a positional measurement of 3 micro-meters on 
a camera base of 1 meter. 

The problems with the second method concern noise, image distortion and 
miss-recognition. 

In field trials with televised baseball and football games, previous systems encountered the 
following specific, major problems. 
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In a typical sport, such as football or baseball, close up shots are taken with long focal 
length cameras operating at a distance of up to several hundred yards from the action. Both of 
these sports have sudden action, namely the kicking or hitting of a ball, which results in the game 
changing abruptly from a tranquil scene to one of fast moving action. As the long focal length 
cameras react to this activity, the image they record displays several characteristics which render 
motion tracking more difficult. For example, the motion of the image may be as fast as ten pixels 
per field. This will fall outside the range of systems that examine pixel windows that are less than 
10 by 10 pixels. Additionally, the images may become defocused and suffer severe motion 
blurring, such that a line which in a static image is a few pixels wide, blurs out to be 10 pixels 
wide. This means that a system tracking a narrow line, suddenly finds no match or makes 
assumptions such as the zoom has changed when in reality only fast panning has occurred. This 
motion blurring also causes changes in illumination level and color, as well as pattern texture, all 
of which can be problems for systems using pattern based image processing techniques. Camera 
motion, even in as little as two fields, results in abrupt image changes in the local and large scale 
geometry of an image. An image's iUumination level and color are affected by camera motion as 
well. 

2. Moving objects 

Sports scenes generally have a number of participants, whose general motion follows some 
degree of predictability, but who may at any time suddenly do something unexpected. This means 
that any automatic motion tracking of a real sports event has to be able to cope with sudden and 
unexpected occlusion of various parts of the image. In addition, the variety of uniforms and poses 
adopted by players in the course of a game, mean that attempts to follow any purely geometric 
pattern in the scene have to be able to cope with a large number of occurrences of similar 
patterns. 

3. Lens distortion 

All practical camera lenses exhibit some degree of geometric lens distortion which changes 

3 



SUBSTITUTE SHEET (RULE 25) 



WO 97/00494 PCT/US96/10164 

the relative position of objects in an image as those objects move towards the edge of an image. 
When l/10th of a pixel accuracy is required, this can cause problems. 

4. Noise in the signal 

Real television signals exhibit noise, especially when the cameras are electronically boosted 
to cover low light level events, such as night time baseball. This noise wreaks havoc with image 
analysis techniques which rely on standard normalized correlation recognition, as these match 
pattern shapes, irrespective of the strength of the signal. Because noise shapes are random, in the 
course of several hundred thousand fields of video (or a typical three hour game), the chances of 
mistaking noise patterns for real patterns can be a major problem. 

5. Field-to-field interlace 

Television images, in both NTSC and PAL standards, are transmitted in two vertically 
interlaced fields which together make up a frame. This means that television is not a single stream 
of images, but two streams of closely related yet subtly different images. The problem is 
particularly noticeable in looking at narrow horizontal lines, which may be very evident in one 
field but not the other. 

6. Illumination and color change 

Outdoor games are especially prone to illumination and color changes. Typically, a 
summer night baseball game will start in bright sunlight and end in floodlight darkness. An 
illumination change of a factor of more than two is typical in such circumstances. In addition the 
change from natural to artificial lighting changes the color of the objects in view. For instance, at 
Joe Robbie Stadium in Florida, the walls appear blue under natural lighting but green under 
artificial lighting. 

7. Setup differences 

Cameras tend to be set up with small but detectable differences from night to night. For 
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instance, camera tilt typically varies by up to plus or minus 1%, which is not immediately obvious 
to the viewer. However, this represents plus or minus 7 pixels and can be a problem to typical 
templates measuring 8 pixels by 8 pixels. 

The prior art contains a number of proposals for automatic motion tracking in video 
sequences. 

Rosser et al., in US patent no. 5,264,933, utilized well known methods of image 
processing, such as template correlation combined with pyramid techniques. See, "IEEE 
Proceedings", May 1979 and "Pattern Recognition & Image Processing", Vol 67, Num. 5, p P 
705-880. It was thought that this would be sufficient for all field to field tracking of motion once 
the pattern had been recognized. This method works well in limited, controlled environments. 
However, such standard algorithms are insufficient for the demands of day to day broadcast 
environments and need to be improved upon for general, wide spread commercial use of video 
insertion systems. The present invention is an improvement over this prior technique. 

Luquet et al., in US patent no. 5,353,392, propose using either camera sensor information 
or landmarks in the scene, and relying on mathematical modeling filtered over time to eliminate 
uncertainties in either motion or position. However, this method presupposes an evolutionary 
image sequence with a high degree of correlation between each image. The abrupt changes in 
action of many major sports, and the subsequent abrupt and non-evolutionary changes in both 
camera and image parameters makes it unlikely that their algorithms will be successful in tracking 
through those abrupt changes which are often the moments of most interest to the viewer. 

Hanna et al., in PCT application PCT/US92/07498 (WO 93/06691), propose a two step 
approach to tracking in which a set of widely distributed set of landmarks are followed using 
normalized correlation on a Laplacian decimated pyramid of the video field. This coarse location 
is used as a guide for performing an affine precise alignment on a "locator pattern" within the 
image. The process is described in detail in "Hierarchical Model-Based Motion Estimation" in the 
Proc. European Conference on Computer Vision, 1992, pp 237-252, by Bergen et al. It uses an 
iterative process of calculating image flow by subtracting a reference locator pattern from the 
current location predicted by the course alignment. It then uses that image flow to warp the 
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locator pattern to a more precise nation. Although capable of very high accuracy, this method 
requires a starting point that is accurate to less than a pixel. Even then it is prone to error in the 
presence of noise or moving objects, therefore unsuited for day to day general use. One approach 
to locating landmarks in a scene employing dissimilar templates is described in our co-pending 
U.S. Patent Application No. 08/381,088 filed January 31, 1995 entitled "Live Video Insertion 
System", the entire contents of which are incorporated herein by reference. 

The problem addressed by the present invention is the difficulty of providing a robust yet 
high precision tracking algorithm capable of following motion in a stream of real, noisy video 
images, thereby enabling live video insertion systems capable of general commercial i 



. use. 



ma 



Summary of the Invention 

The invention comprises an improved method for tracking motion from field to field 
sequence of related video images. Termed adaptive-geographic-hierarchical tracking with color 
based landmark occlusion, the method allows robust, high precision motion tracking in a stream 
of real, noisy video images. 

The preferred method uses template correlation with zoom insensitive templates, such as 
edges, to follow a group of pre-designated landmarks or some subset of a group within a scene. 
Template correlation of landmarks provides raw position information used to follow the motion 
of a scene. Typically, the landmarks used may be parts of the structure in a ball park or markings 
on a field of play. Creating an ideal mathematical formulation of the scene to be tracked is a key 
part of the tracking algorithm. This ideal mathematical representation is referred to as the 
reference array and is simply a table of idealized x,y coordinate values. The images associated 
with the array are for operator convenience. Current images or scenes are related to this reference 
array by a set of warp parameters which define the mathematical transform that maps points in 
the current scene to the reference array. In the simple case in which rotation is ignored or kept 
constant the current image is mapp,d to the reference array by the following equations: 

x'=a+bx 

y'=d+by 
6 
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where x' and y' are the coordinates of a landmark in the current scene, x and y are the coordinates 
of the same landmark in the reference array and b is the magnification between the reference array 
and the current scene, a is the translation in the x direction and d is the translation in the y 
direction between the reference array and the current scene. 

The essence of adaptive, geographic hierarchical tracking is paying most attention to 
landmarks which are found at or close to their anticipated model derived positions. 

The first step is to obtain an accurate velocity prediction scheme to locate the anticipated 
model derived position. Such a scheme estimates, via the warp parameters from the previous field 
or scene, where the landmarks in the current image should be. The primary difficulty with velocity 
prediction in interlaced video is that from field to field there appears to be a one pixel y 
component to the motion. The present invention handles this by using the position from the 
previous like field, and motion from the difference between the last two unlike fields. 

Having predicted where in the current image the landmarks should be, template 
correlations over a 15 by 15 pixel region are then performed centered on this predicted position. 
These correlation patterns are then searched from the center outward looking for the first match 
that exceeds a threshold criteria. Moreover, each landmark has a weighting function whose value 
is inversely proportional to the distance the landmark is away from its anticipated model derived 
position. When calculating the new warp parameters for the current scene, each landmark's 
current position is used weighted by this function. This gives more emphasis to landmarks which 
are closer to their predicted positions. A further step, necessary to compensate for camera 
distortion as the scene moves, is to dynamically update .the reference array coordinates of the 
landmarks based on their current locations. This updating is done only on good landmarks, and 
is itself heavily weighted by the distance error weighting function. This adaptive reference array 
allows very accurate tracking of landmarks even as they pass through lens and perspective 
distortions. The danger in having an adaptive reference array is that it may get contaminated. This 
danger is mitigated by having three sets of reference coordinates, which are referred to as the 
code, game and tracking reference coordinates. When the system is initially loaded, the code 
reference coordinates are set to the original reference coordinates. The game and tracking 
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coordinates are initially set equal to the code reference coordinates. Once the system locates a 
scene and begins tracking, the tracking coordinates are used. However, each time a scene cut 
occurs, the tracking coordinates are automatically reset to the game reference coordinates. At any 
time the operator may choose to set the current tracking coordinates equal to the game reference 
coordinates or to set the game reference coordinates back to the code reference coordinates. This 
scheme allows for adaptive reference updating with operator override capability. 

The final element in the tracking scheme is a method of determining when a landmark is 
obscured by some object, so as to avoid spurious data in the system. A color based occlusion 
method is used in which a set of sensor points in a pattern around where a landmark is found are 
examined and if they are found to from the colors expected in those regions, the landmark 
is deemed to be occluded and not used in further calculations. The sensor points from good 
landmarks are used to update the reference values for expected colors of the sensor points so that 
the system can accommodate changing conditions such as the gradual shift from sunlight to 
artificial light during the course of a broadcast. 

This strategy of adaptive, hierarchical tracking has proved to be a means of high precision 
and robust tracking of landmarks within video sequences even in the noisy, real world 

environment of live broadcast television. 



Brief Descripti on of the Drawing 

Fig. 1 is a schematic representation showing a reference video image of a scene. 

Fig. 2 is a schematic representation showing a live video image of the reference video 
image in Fig. 1. 



Fig. 3 is a table illustrating the elements of a typical representation of a reference 



array. 



Fig. 4 is illustrates a schematic representation of field number versus y image position in 
an interlace video field. 



SUBSTITUTE SHEET (RULE 26) 



WO 97/00494 PCT/US96/10164 
Fig. 5a illustrates a cross-sectional view of zero mean edge template. 

Fig. 5b illustrates a plan view of a zero mean edge template 

Fig. 6 illustrates a correlation surface. 

Fig. 7 illustrates a measured and predicted position on a surface. 

Fig. 8 illustrates a schematic flow diagram of how a track, reference, and code hierarchy 
of reference arrays is used to manage an adaptive reference array. 

Fig. 9 illustrates a schematic view of landmarks and their associated sensor points used 
for color based occlusion. 



Detailed Pes O ption of the Preferred FmhnHimgnt 

During the course of this description like numbers will be used to identify like elements 
according to the different figures which illustrate the invention. 

Adaptive-geographic-hierarchical tracking with color based occlusion is a method which 
allows for the robust, high precision tracking of motion from field-to-field in a sequence of related 
video images in the presence of noise as experienced in a typical broadcast. As its name suggests, 
there are several parts to the method. To understand the invention, it is easiest to take component 
parts and look at each in turn. 

The reason for doing motion tracking is to allow the electronic insertion of images into 
live video signals so that they appear to be part of the original scene, primarily for the purposes 
of advertising, even when the camera zooms, pans, tilts, or otherwise alters the appearance of the 
scene in size or perspective, as dest. -oed in US Patent 5,264,933 by Rosser et al. 

Referring to figure 1, motion tracking of video images which allow seamless insertion as 

9 
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practiced by this invention, starts with a reference array 10 of a scene in which insertions are to 
be placed. Although having an actual image is a useful mental aid, this reference array is nothing 
more than a set of idealized x,y coordinate values which represent the position of a number of key 
landmark sets 16 and 18 within reference array 10. A typical table is shown in figure 3 
illustrating the listing of x, or horizontal coordinates 3 1, and the y, or vertical coordinate positions 
33. The positions 31 and 33 of key landmark sets 16 and 18 are used both as references against 
which motion can be measured * 5 in relation to which insertions can be positioned. A typical 
reference array 10 of a baseball scene from a center field camera will consist of the locations of 
features such as the pitchers mound 12, the back wall 14, vertical lines 1 5 between the pads which 
make up the back wall 14, and the horizontal line 17 between the back wall and the field of play 
on which the horizontal set of landmarks 18 are set. 

The current image or scene 20 is the field from a video sequence which is presently being 
consuiered. Locations of key features or landmark sets 16 and 18 from reference array 10 also 
are mdicated in current image 20 as measured positions 26 and 28. Measured positions 26 and 
28 are related to corresponding reference array landmark locations from sets 16 and 18 by a set 
of warp parameters which define a mathematical transform that most accurately maps the position 
ofpointsincurrent image 20 to the position of points in reference array 10. Such mappings are 
well known mathematically. See, "Geometrical Image Modification in Digital Image Processing", 
W.K. Pratt 2nd Edition, 1991, John Wiley and Sons, ISBN 0-471-85766. 

Tracking the view from a fixed television camera, especially one with a reasonably long 
focal length as in most sporting events, can be thought of as mapping one two-dimensional surface 
to another two-dimensional surface. A general mathematical transform that accomplishes such 
a mappmg allowing for image to image translation, zoom, shear, and rotation is given by the 

following six parameter model 

x ; =a+jbx+cy 



y /= d+ex+fy 
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where 

x and y are coordinates in reference array 10, 

x' and y- are the transformed coordinates in current image 20, 

a is the image translation in the x direction, 

b is the image magnification in the x direction, 

c is a combination of the rotation, and skew in the x direction, 

d is the image translation in the y direction, 

e is a combination of the rotation, and skew in the y direction, and 

f is the image magnification in the y direction. 

The tracking algorithms and methods discussed herein can be used with the above 
transformation as well as other n,o , general transformations. However, experience has shown 
that w,th a dynamically updated reference array, a simpler x,y mapping function which assumes 
no shear or rotation will suffice. Thus, in the simple case in which rotation is ignored or kept 
constant (c = e = 0) and the magnification in the x and y directions is the same (b = f) the position 
of po,nts in current image 20 are mapped to the position of points in reference array 10 using the 
following equations: 

x 7 =a+jbx 
y f =d+by 

where x' and f are coordinates of a landmark in current image 20, x and y are coordinates of the 
same landmark in reference array 10, b is the magnification between reference array 10 and 
current unage 20, a is the translation in the x direction, and d is the translation in the y direction 
Th* amplified mapping scheme is used because experience has shown it to be both robust and 
capable of handling the limited she ,, rotation, and perspective distortion present in television 
sports broadcasts when a dynamically updated reference array is used. 

Motion tracking is the method of measuring positions of landmark sets 26 and 28 in 
current image 20 and using these measurements to calculate the warp parameters a, d and b as 
defined by the equations above. An important part of adaptive geographic hierarchical tracking 
» the concept of assigning a weight to each landmark. Weights are assigned, in inverse 
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proportion, according to the distance each landmark is detected away from where it is expected 
or predicted to be found. The closer a landmark is found to where it is predicted to be, the 
greater the weight given to that 1, : ; rimark in the calculation of the warp parameters Unking the 
positions in current image 20 to the positions in reference array 10. 

The first step is predicting where the landmarks 26 and 28 should be in current image 20 
This is done by analyzing the landmark positions in the three previous fields. The previous 
position and velocity of a landmark derived from the previous model is used to estimate where 
the landmark will appear in the current image 20. The position and velocity calculations are 
complex in that both the current standard methods of television transmission, NTSC and PAL 
are sent in two vertically interlaced fields. Thus, alternate horizontal scans are included k 
separate fields, customarily referred to as odd and even fields. In the NTSC system, each field is 
sent in l/60th of a second (16.6 msecs), making a combined single frame every l/30th of a 
second. 

One important practical consideration in the velocity estimations is that the x and the y 
positions in the previous fields (-1, -;. sad -3) that are used in the velocity estimations are not the 
measured positions, but the positions calculated using the final warp parameters derived in each 
of those fields. That is, in each field, x and y positions are measured for each landmark. All of the 
landmarks are then used to derive a single set of warp parameters a, b and d giving the mapping 
between the current and the reference array. That single set of warp parameters is then used to 
project the reference array coordinates 10 into the current image 20, giving an idealized set of 
landmark positions in the current image. It is this idealized set of landmark positions in each field, 
referred to as the model derived positions, that are used in the velocity predictions. 

As illustrated in figure 4, the current y or vertical position of a landmark is predicted from 
the previous three fields. The y position in the current field (field 0) is predicted by measuring the 
y component of velocity as the difference between the landmark's model derived position in field 
-1 and field -3, which are "like" fields in that they are both either odd or even. The y velocity 
component is then added to the model derived y position in field -2, which is the previous field 
"like" the current field, to arrive at an Ornate of where to find that landmark in the current field 
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The prediction in the x direction could use the same algorithm or, since there is no 
interlace, the x direction calculation can be simpler and slightly more current. In the simpler 
scheme, the x component of the velocity is calculated as the difference between the landmark's 
model derived position in field -1 and its model derived position in field -2. This difference is then 
added to the model derived position in field -1 to arrive at an estimate of where to find that 
landmark in the current field. 



Having predicted the mos likely position of all the landmarks in the current image, the 
positions of the landmarks are then found by doing a correlation of an 8 by 8 pixel template over 
a 15 by 15 pixel region centered at the predicted position. Correlation or template matching is a 
well known technique, and in its standard form is one of the most fundamental means of object 
detection. See, Chapter 20, "Image Detection and Recognition of Digital Image Processing" by 
W.K. Pratt (2nd Edition, 1991, John Wiley and Sons, ISBN 0^71-85766). Unlike more standard 
methods of correlation or template matching in which the template is made to closely resemble 
the part of the scene it is being used to find, the templates in the present invention are synthetic 
idealized both in shape and value, and are "zero-mean". For instance, in tracking a football goal 
post upright, rather than use a portion of the goal post taken from the image, the template 54 used 
is an edge of uniform value made from a negative directed line 56 and a positive directed line 58, 
and the sum of the values in the 8 by 8 template is equal to zero as shown schematically m 
cross-section in figure 5a and in plan view in figure 5b. This template has the advantages of being 
zoom independent and will give a zero value on a surface of uniform brightness. The technique 
is not limited to 8 by 8 pixel temp' ,es, nor is me region over which they are correlated limited 
to 15 by 15 pixel regions. Further, this technique is not limited to zero mean templates either. 
In circumstances where only vertical and horizontal lines and edges are being tracked it is possible 
to reduce computation by having (1 x n) correlation surfaces for following the horizontal detail, 
and (n x 1) correlation surfaces for following the vertical detail where n is any reasonable number, 
usually in the range of 5-50 pixels. 

The idealized, zero-mean edge template 54 is correlated over a 15 by 15 pixel region of 
the current image or some amplified, filtered and decimated replica of it to produce a correlation 
surface 60 as shown schematically in figure 6. This correlation surface 60 consists of a 15by 15 
array of pixels whose brightness correspond to the correlation of the image against the template 
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when centered at that position. Typically, an edge template 54 correlated over a region of an 
.mage containing a line will give both a positive going line response 66, indicating a good match 
and a corresponding negative going line 67 indicating a mismatch. This mismatch line 67 can be 
useful in that its position and distance away from the positive going match line 66 give a measure 
of the with of the line and whether it is brighter or darker than its surroundings. In addition, there 
wdl be other bright pixels 68 on the correlation surface 60 corresponding to bright edge like 
features in the cun-ent image. 

A guiding principle of ,he adaprive-geographichierarchical tracking method is ,o focus 
on landmark* and the correlation peaks indlciltil , s potMtial lmdmaI|[s ^ m ^ ^ 
they - expected to be. Raher than just looking for a peak anywhere on the ! 5 by 13 cordon 
surface 60, th«e pane™ are searched from the c M er outward. The staphs and ve^ effective 
way of doing this is to firs, look a„,e cemral nine pixel valuB ta „, centnl , by , ^ ' 
64. If any of these pixels has a correlation value greater than a threshold then it is assumed that 
the pnte, represents the landmark being sough, and no further instigation 0 f the correlation 
surface is done. The threshold is usually fifty percent „ f the usual tadmark correction 
anticpated. This 3 by 3 initia! search ano™ motion tracking even in the presence of nearby 
obKctt that by their brightness or shape might confuse the iandmark correlation such as when the 
P«.l marked 68 had been brighter than the pixds in the line 66. Once the pixel with the peak 
bnghtness is fou»4 an estimate of me sub pixel position is found using the well known method 
of reacting a triar^e as discussed in co-pending „.S. Pat. Appl. No. 08/381,088 There 
are other sub pixel position estimating raahods th>t my ^ ^ ^ 

anvestothedat, However, experience has sh„™ tha. the tiiangular m«hod describe in cc- 
penduig U.S. Pat. Appl. No. 08/381,088 to be most effective. 

An alternative method is to multiply th. correlation surface by a function which gives 
greater weight I0 pixels at ^ r ^ ^ ^ ^ 

h-found -top-ha,.. and Guassian like shapes to be most effective. This new weighted surface 
■s then searched for peak values and tee are used to give the location a, which the subphcel 
^natesare based. When using th. weighted^ for andingpeaks.it is still necessary ,„ use 
» unweighted correction .rface for subpixe, peak estimation using the mettod of completing 
the mangle as described in co-pending U.S. Pat. Appl. No . 0MS ,, 0 8 8 or other like algorithms 
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It is also important to check that the peak from the weighted surface is in fact a local peak on the 
unweighted surface before doing the interpolation. If it is not then no interpolation is performed 
and the uninterpolated value is used. 



In addition, each landmark found in a scene has an error weight associated with it based 
on its distance from where it is expected to be. Referring now to figure 7, the calculation of this 
error weight is based on the predicted position in the image 70, at the coordinates xp, yp and the 
measured position in the image 72, at the coordinates xm, ym, using the general equation: 



ErrorWeight- 



h+(i( (xp-xm) ^+ (yp-ym) k ) ) 1 



where g, h, i, j, k, and 1 are numerical constants chosen to vary the strength of the weighting 
function. 



In the preferred embodiment the parameters of the equation are: 



Errorfteight= 



1.0 



1 . 0+ ( (xp-xm) 2 + (yp-ym) 2 ) 1 -° 



although in special circumstances, each of the parameters might have a different value to change 
the emphasis of the weighting. For instance, numerical constants i and j may be varied to provide 
a function which stays constant for a short distance and then drops rapidly. 



This error weight is then used in the calculation of the warp parameters which maps points 
in the current image 20 to the positions in the reference array 20. In the preferred embodiment this 
calculation is a weighted least mean squares fit using the following matrix: 



£ ( ci-Cl ) £ ( nx-Cl )L{nycl) 
£ ( nx-Cl ) £ ( /ix- nx) E (nx-ny) 
T,(nyCl)l:{ nx-ny) E ( nyny) 



T.{C1-C2) 
E(nx-C2) 
T(nyC2) 
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where 

CI = nx-Err orWel gh t-xp+nyErrorWei gh fyp 



and 

C2=nx-ErrorWei gh t-xm+nyErrorWei gh fym 



In the case of purely horizontal landmarks, nx=0 and ny=l and in the case of purely 
vertical landmarks nx=l and ny=0. Li the more general case, nx and ny are the direction cosines 
of vectors representing the normal to the landmarks predominant direction. 

The adaptive part of the motion tracking scheme is necessary to allow for camera 
distortion. It also allows the system to compensate for small discrepancies between the stored 
idealized reference array and the actual scene as well as allowing the system to handle small slow 
rotation and/or shear. It further allows the system to handle any small and slowly occurring 
distortions. This adaptation is done by dynamically updating the reference array coordinates based 
on their current locations. In the present invention the adaptive part of the motion tracking is 
made stable by the following criteria: 1) being very careful when it is allowed to occur, 2) 
choosing which landmarks are allowed to participate based on how confident the system is that 
said landmarks are good; and 3) h,vmg the whole calculation heavily weighted by the distance 
error weighting function. In addition, the reference array is reset after any scene cuts. 

In the preferred embodiment the dynamic updating of the reference coordinates is started 
after six fields of tracking and is only done on landmarks which have not been flagged by any 
occlusion checks and have correlation values greater than 20% and less than 200% of expected 
reference values, though different values may be used for all these parameters. 

The measured landmark positions are back projected to the positions in the reference array 
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using the warp parameters calculated by all the good landmarks in the current field using the 
equations; 



Xnr= (Xm-a) /b 



Ynr= ( Ym-d) /b 



Xr=X0r+- (ErrozWeight) 2 * (Xnr-XOr) 
Yr=Y0r- {ErrozWeight) 2 - ( Ynr-YOr) 



where: 

Xm is the measured x coordinate of the landmark, 
Ym is the measured y coordinate of the landmark, 
a is the horizontal translation warp parameter, 
d is the vertical translation warp parameter, 
b is the magnification warp parameter, 

Xnr is the calculated x coordinate of a proposed new reference point based on this field's 
data, 

Ynr is the calculated y coordinate of a proposed new reference point based on this field's 
data, 

XOr is the x coordinate of the old reference point prior to update, 
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YOr is the y coordinate of the old reference point prior to update, 

Xr is the x coordinate put into the table as the new reference point, and 

Yr is the y coordinate put into the table as the new reference point. 

It is also possible to use separate tracking reference arrays for odd and even fields to 
improve the tracking performance with interlace video. Because of the potentially unstable nature 
of the adaptive reference array, the preferred embodiment has three related reference arrays, 
referred to as the: 

CODE REFERENCE 
GAME REFERENCE, and 
TRACKING REFERENCE 

The schematic flow diagram in figure 8 illustrates how these three references are used. At start 
up, when the initial system is loaded, all three references are set to be the same 

CODE REFERENCE = GAME REFERENCE = TRACKING REFERENCE 

which is to say that the x and the y coordinates of the landmarks in each of the reference arrays 
are set to be the same as the coordinates of the landmarks in the code reference array. 

At run time, when the image processing is done, the three reference arrays are used in the 
following manner. The game reference is used in search and verify mode and in tracking mode 
the tracking reference is used. 

Initially the tracking reference array is set equal to the game reference array. In the 
preferred embodiment this occurs on the first field in which the tracking is done. In subsequent 
fields the tracking reference is modified as detailed above. If separate tracking reference arrays 
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are being used for odd and even fields they would both initially be set to the game reference array. 



At any time during the tracking mode, the operator may elect to copy the cuirent tracking 
references into the game reference using standard computer interface tools such as a screen , 
keyboard, mouse, graphic user interface, trackball, touch screen or a combination of such devices. 
This function is useful at the start of a game. For instance, an operator may be setting up the live 
video insertion system to perform insertions at a particular stadium. The code reference 
coordinates have landmark positions based on a previous game at that stadium but the position 
of the landmarks may have been subtly altered in the intervening time. The code reference, 
however, remains good enough for search and tracking most of the time. Alternatively, by waiting 
for a shot, or having the director set one up prior to the game, in which all the landmarks are clear 
of obstruction, and allowing for the adjustment of the tracking reference to be completed, a more 
accurate game reference for that particular game can be achieved. 

At any time, in either the tracking or search mode, the operator can elect to reset the game 
reference to the code reference. This allows recovery from operator error in resetting the game 
reference to a corrupted tracking reference. 

An important part of the adaptive reference process is restricting the updating to 

landmarks which are known to be un-occluded by objects such as players. The method used for 

this landmark occlusion detection in the preferred embodiment is color based and takes advantage 

of the fact that most sports are playe-a on surfaces which have well defined areas of fairly uniform 

color, or in stadiums which have substantial features of uniform color, such as the back wall in 

a baseball stadium. Each landmark 90 as shown in figure 9, has sensor points 92 associated with 

it. These sensor points 92, which in the preferred embodiment vary from 3 to 9 sensor points per 

landmark 90, are pixels in predetermined locations close to, or preferably surrounding the 

landmark they are associated with. More importantly, the sensor points are all on areas of 

reasonably uniform color. The decision on whether the landmarks are occluded or not is based 

on looking at the sensor points and measuring their deviation from an average value. If this 
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deviation exceeds a pre-set value, the landmark is presumed to be occluded. Otherwise it is 
available for use in other calculations, such as the model calculations and the reference array 
updating. 

Both the method of dealing with vertical interlace and the use of idealized landmark 
positions in each of the previous fields are vital and innovative. 



The present invention possesses several advantages over the prior art known to applicant. 

First, the present invention is more robust than any of the known prior art systems in that 
it is capable of tracking motion from field-to-field in a typical broadcast environment while 
maintaining a high precision of accuracy. 



Second, the present inventi- a is able to handle abrupt changes within a sequence of images 
smoothly. Such changes indicate events like fast action or a scene cut and are often of the most 
interest to a viewer. 

Third, the present invention does not rely on sensor data from the camera itself. Thus, the 
invention is impervious to camera jitter or motion during a broadcast. 

Fourth, since only the video image is used, the system can be located anywhere in the 
broadcast stream between the event and the viewer. 

Finally, the present invention can achieve reliable results without requiring preprocessing 
or a starting point accurate to less than a pixel. 



While the invention has been described with reference to the preferred embodiment 
thereof, it will be appreciated by those of ordinary skill in the art that various modifications can 
be made to the structure and function of the individual parts of the system without departing from 
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I claim: 

1 . A method for tracking motion from field to field in a sequence of related video 
images that are scanned by a video system, the method comprising the steps of: 

a) establishing an array of idealized x and y coordinates representing a reference array 
having a plurality of landmarks where each landmark has unique x and y coordinates; 

b) mapping x and y coordinates in a current image to said x and y coordinates in said 
reference array; and, 

c) predicting the future location of said landmark coordinates, x* and y', based on a 
detected change of position of said landmark from the previous field, 

wherein prediction errors due to changes between two successive fields are minimized. 

2. The method of claim 1 wherein said mapping is achieved according to the 
following relationships: 

x'=a+jbx+cy 
y ! =d+ex+fy 



where: 

x is a horizontal coordinate in the reference array, 
y is a vertical coordinate in the reference array, 
x' is a horizontal coordinate in the current scene, 
Y is a vertical coordinate in be current scene, 

a is a warp parameter for horizontal translation of the object in the x direction, 
b is a warp parameter for magnification between the reference array and the current image 
in the x direction, 

c is a warp parameter for a combination of rotation and skew in the x direction, 

d is a warp parameter for vertical translation of the object in the y direction. 

e is a warp parameter for a combination of rotation and skew in the y direction, and 
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f is a warp parameter for magnification between the reference array and the current image 

in the y direction. 



3 . The method of claim 2 wherein said video images are vertically interlaced where 
images from field to field alternate between like and unlike fields. 

4. The method of claim 3 wherein said predicting the future location of said landmark 
coordinates, x' and y\ for said interlaced video images is based on a detected change of position 
of said landmark from the previous like field. 

5. The method of claim 4 further comprising the steps of: 

d) searching for one of said landmarks in said current image by means of correlation 
using a template where the search is conducted over a substantial region spanning the predicted 
location of said landmark; 

e) multiplying the result- of said correlation search in step (d) by a weighting function 
giving greater weight to correlations closer in distance to the predicted location of said landmark 
to yield a weighted correlation surface; 

f) searching said weighted correlation surface for its peak value. 

6. The method of claim 5 further comprising the steps of: 

g) determining new warp parameters a,b,c,d,e and f for a current image based on said 
landmark's current position in a current image weighted by said weighting function for that 
landmark, 

wherein emphasis is given to landmarks which are closer to their predicted 

position. 

7. The method of claim 6 wherein said weighting function comprises the following 
relationship: 
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ErrorWel ght- E 

(xp-xm) j +(yp-ym) k ) ) 1 



where: 

g,h,i j,k, and 1 are numerical constants; 
xp is the predicted x coordinate location of said landmark; 
xm is the measured x coordinate position of said landmark; 
yp is the predicted y coordinate location of said landmark; and, 
ym is the measured y coordinate position of said landmark. 

8. The method of claim 7 further including the step of: 

h) updating said landmark locations in said reference array according to the location 
of said landmarks in said current image, 

wherein said updating is performed based upon well identified landmarks and 
according to said landmark weighting function. 

9. The method of claim 8 further comprising the step of 

i) establishing three types of reference arrays prior to broadcast including; 

i) a code reference array having landmark coordinates equal to said reference 
landmark coordinates, 

ii) a game reference array having landmark coordinates initially set equal to said 
code reference array coordinates, and, 

iii) a tracking reference array having landmark coordinates initially set equal to said 
code reference array coordinates. 



10. The method of claim 9 further comprising the steps of: 
j) changing said tracking reference array of coordinates during a broadcast; and, 
k) resetting the tracking reference array of coordinates to said game reference array 
of coordinates after a scene cut. 
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1 1 . The method of claim 1 0 wherein said video system is controlled by an operator and 
said method further comprises the step of: 

1) selectively choosing to set said current tracking reference array of coordinates 
equal to said game reference array of coordinates or to set said game reference array of 
coordinates back to said code reference array of coordinates, 

wherein said operator can update or override the game or tracking reference array of 
coordinates. 

12. The method of claim 1 1 further comprising the steps of: 

m) establishing a set of sensor points in a pattern around the location of each said 
landmark said sensor points being able to detect changes in color and illumination; 

n) determining if said sensor points are different in color or illumination from the 
expected color or illumination; and, 

o) excluding said landmark from future calculations if said color or illumination is 
substantially different from what was expected, 

wherein said landmark is deemed to be occluded if said color or illumination at said sensor 
points is substantially different from the expected color or illumination. 

13. The method of claim 12 wherein said correlation template is a 15 by 15 pixel 
window. 

14. The method of claim 1 wherein said mapping is achieved according to the 
following relationships: 

x'=a+bx 
y f =d+by 

where: 
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x is a horizontal coordinate in the reference array, 
y is a vertical coordinate in the reference array, 
x' is a horizontal coordinate in the current scene, 
y' is a vertical coordinate in the current scene, 

b is a warp parameter for magnification between the reference array and the current image, 
a is a warp parameter for horizontal translation of the object in the x direction, and, 
d is a warp parameter for vertical translation of the object in the y direction. 

15. The method of claim 4 further comprising the steps of: 

P) searching for one of said landmarks in said current image by means of correlation 
using a template where the starting point of the search is substantially centered at the predicted 

location of said landmark; 

q) performing said search beginning from said predicted location and proceeding 
outward looking for a match; and, 

r) discontinuing said search for said landmark when said match exceeds a threshold 

value. 



16. The method of claim 6 wherein said weighting function comprises the following 
relationship: 

ErrorWei ght- — — 1 • 0 

l-0 + ( (xp-xm) 2 +(yp-yni)2)i-o 



where: 

xp is the predicted x coordinate location of said landmark; 
xm is the measured x coordinate position of said landmark; 
yp is the predicted y coordinate location of said landmark; and, 
ym is the measured y coordinate position of said landmark. 



17. A system for tracking motion from field to field in a sequence of related video 
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images that are scanned by a video system comprising: 

scanning means for establishing an array of idealized x and y coordinates representing a 
reference array having a plurality of landmarks where each landmark has unique x and y 
coordinates; 

mapping means for mapping x and y coordinates in a current image to said x and y 
coordinates in said reference array; and, 

predicting means for predicting the future location of said landmark coordinates, x* and 
y\ based on a detected change of p -ution of said landmark from the previous field, 

wherein prediction errors due to changes between two successive fields are minimized. 

1 8. The system of claim 1 7 further comprising: 

search means for searching for one of said landmarks in said current image with a 
correlation template substantially centered at the predicted location of said landmark wherein said 
search is performed beginning from said predicted location and proceeding outward looking for 
a match and discontinued when said match exceeds a threshold value. 

19. The system of claim 1 8 further comprising: 

weighting means for assigning each landmark a weighting function whose value is 
inversely proportional to the distance away said landmark is from its predicted position. 

20. The system of claim 1 9 further comprising: 

updating means for updating said landmark locations in said reference array according to 
the location of said landmarks in said current image wherein said updating is performed based 
upon well identified landmarks and according to said landmark weighting function. 

2 1 . The system of claim 20 further comprising: 

means for establishing three types of reference arrays prior to broadcast including; 

i) a code reference array having landmark coordinates equal to said reference 
landmark coordinates, 
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ii) a game reference array having landmark coordinates initially set equal to said 
code reference array of coordinates, and, 

iii) a tracking reference array having landmark coordinates initially set equal to said 
code reference array of coordinates. 

22. The system of claim 21 further comprising: 

operator input means for changing said tracking reference array of coordinates during a 
broadcast and resetting the tracking reference array of coordinates to said game reference array 
of coordinates after a scene cut. 

23 . The system of claim 22 further comprising: 

sensor means for establishing a set of sensor points in a pattern around the location of each 
landmark wherein said sensor points are able to detect changes in color and illumination for 
determining if said sensor points are different in color or illumination from the expected color or 
illumination. 

24. The system of claim 23 further comprising: 

occlusion means for excluding a landmark from future calculations if said color or 
illumination around said landmark is substantially different from what was expected. 
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AMENDED CLAIMS 

[received by the International Bureau on 16 October 1996 (16 10 96)- 
original claim 1 amended; new claim 25 added; remaining claims unchanged' (2 pages)] 



I claim: 

1. A method for tracking motion from field to field in a sequence of related video 
images that are scanned by a video system, the method comprising the steps of: 

a) establishing an anray of idealized x and y coordinates representing a reference array 
having a plurality of landmarks where each landmark has unique x and y coordinates; 

b) mapping x and y coordinates in a current image to said x and y coordinates in said 
reference array by means of a single mapping function for the camera related motion of the entire 
image; and, 

c) predicting the future location of said landmark coordinates, x* and y\ based on a 
detected change of position of said landmark from the previous field, 

wherein prediction errors due to changes between two successive fields are minimized. 

2. The method of claim 1 wherein said mapping is achieved according to the following 
relationships: 

x'=a+bx+cy 
yt-d+ex+fy 

where: 

x is a horizontal coordinate in the reference array, 
y is a vertical coordinate in the reference array, 
x* is a horizontal coordinate b the current scene, 
y* is a vertical coordinate in the current scene, 

a is a warp parameter for horizontal translation of the object in the x direction, 
b is a warp parameter for magnification between the reference and the current image in the 
x direction, 

c is a warp parameter fox a combination of rotation and skew in the x direction, 

d is a warp parameter for vertical translation of the object in the y direction. 

e is a warp parameter for a combination of rotation and skew in the y direction, and 
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H) a game reference array having landmark coordinates initially set equal to said 
code reference array of coordinates, and, '«Juaitosaid 

code reference array of coordinates. wwTosaia 

22. The system of claim 21 further comprising: 

operator input means for changing said tracking reference array of coordinates d ■ 
broadcast and resetdng the tracking reference array of co^Sj^ZZZ' 
of coordinates after a scene cut. reference array 

23. The system £ *daim 22 further comprising- 

24. The system of claim 23 further comprising 

occlusion means for excluding a landmark from future calculations if said color or 

25. A method for tracking motion from field to field in a sequence of related video 
images that are scanned by a video svstem *h~ a .. 

»y a viaeo system, the method comprising the steps of: 

a) establishing an array of ide^ized x and y coordinates representing a reference array 
hav^ga^ 

b) ^^"oranat^ 



dete JL 'T" 8 ^ ' UlUrC ' OCati0n ° f ,andm8rk *' - * b-ed on a 

detected change of pos,t,on of said landmark from the previous field 

and using the difference between the previously predicted positions, x' and y f or the 

by n.nmuzmgthewe.ghted error-function between actual positions xandy andthelT 
mapped from the ideated reference array, rx and ry. ^° nS 
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STATEMENT UNDER ARTICLE 19 



The replacement sheets (22 and 28) containing amended claim 1 and new 
claim 25 are the result of receiving the International Search Report from the USPTO. 
Applicant has amended claim 1 in an effort to distinguish the present application from 
the cited reference which was identified as an a X* category reference. 



31 



WO 97/00494 



PCTAJS96/10164 




WO 97/00494 



PCTAJS96/10164 



2/6 




WO 97/00494 



PCT/US96/10164 



3/6 









FIG. 


3 






Tn 

1U 


Lev 


v 

A 


Y 


Til 

TName 


LxT 


(Link ATTRIBUTES) 


0 


1 


36.718 


36.000 


vM7 


9000 


vl 


i 


1 


75.912 


36.000 


vM7 


9000 


vl 


2 


1 


101.269 


36.000 


vM7 


9000 


vl 


3 


1 


179.776 


36.000 


vH7 


9000 


vl 


4 




227.794 


36.000 


vM7 


9000 


vl 


5 


J 


261.520 


36.000 


vM7 


9000 


vl 


6 




297.343 


36.000 


vH7 


9000 


vl 


7 


1 


320.523 


36.000 


vM7 


9000 


vl 


8 


0 


H4.000 


106.448 


hMl 


6148 


h. 


9 


0 


179.000 


106.716 


hMl 


6148 


h. 


10 


0 


240.000 


106.555 


hMl 


6148 


h. 


11 


0 


310.000 


106.747 


hMl 


6148 


h. 


12 


0 


400.000 


106.848 


hMl 


6148 


h, 


13 


0 


4B0.000 


106.724 


hMl 


6148 


h, 


14 


0 


540.000 


106.988 


hMl 


6148 


h. 


15 


0 


619.000 


107.580 


hMl 


6148 


h. 


16 


0 


700.000 


107.748 


hMl 


6148 


h. > 








^33 














FIG. 


4 





















FIELD NUMBER 
y POSITION 



-3 




WO 97/00494 



PCT/US96/10164 



4/6 



FIG. 5 A 









HO 




0 




0 




0 




-10 









FIG. 5B 



1 



54 



56 



7—7- 

/ 
/ 
/ 

/ 
/ 



"A 



58 



FIG. 9 



"90 



92 



97/00494 



PCT/US96/10164 



5/6 



FIG. 6 



\/./ 



-7 



Y/ /, 

V / 



68 



6<K 



62 



7 



V//\ 

// 

'/A 

t 



66 



60 




WO 97/00494 



PCT/US96/10164 



(initial loadinT 



6/6 
FIG. 8 



REFERENCES: TRACK 
GAME = CODE 



RUN TIME) 

EL 



IF SEARCH FOUND 
SOMETHING BUT NEEDS 
CONFIRMATION 




IF VERIFY 
HAS CONFIRMED 
SEARCH 



OPERATOR 



requests) 



SEARCH 




USE GAME 
REFERENCE 
TABLE 


MODE 







VERIFY 




USE GAME 
REFERENCE 
TABLE 




MODE 









TRACK 




MODE 



USE TRACK 
REFERENCE 
TABLE 

INITIALLY 
TRACK = GAME 

UPDATE TRACK 
AFTER 10 FIELDS 



RESET REFERENCES: TRACK = GAME * CODE 



SET GAME - TRACK REFERENCE 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US96/10164 



A. CLASSIFICATION OF SUBJECT MATTER 

IPC(6) :G06K9/00 

US CL : Please See Extra Sheet. 
According to International Patent Classification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 382/103, 104, 105, 107, 173, 181, 190, 209, 217, 218. 219, 220, 221, 232, 236, 238, 251, 252, 253 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) 
APS 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



US. A, 5,001,560 (ERICSSON) 19 March 1991, see 
abstract, figures 1-3, column 1, lines 13-64, column 2, lines 
10-61. 



1, 17 



| [ Further documents are listed in the continuation of Box C. | | See patent family annex. 



" Special categories of cited documents: 

"A* document defining the general slate of the art which is not considered 

to be part of particular relevance 

"E" earlier document published on or after the international filing dote 

*L* document which may throw doubts on priority chunks) or which is 

cited to establish the publication date of another citation or other 
special reason (as specified) 

"O* document referring to an oral disclosure, use. exhibition or other 

means 

"P" document published prior to the international filing date but later than 
the priority dote claimed 



"P later document published after the international filing date or priority 

date and not in conflict with the application but cited to understand the 
principle or theory underlying the invention 

"X* document of particular relevance; the claimed invention cannot be 

considered novel or cannot be considered to involve on inventive step 
when the document ia taken alone 

"Y" document of particular relevance; the claimed invention cannot be 

considered to involve en inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

*&' document member of the same patent family 



Date of the actual completion of the international search 



19 JULY 1996 



Dale of mailing of the international search report 



n aug we 



Name and mailing address of the ISA/US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C, 20231 
Facsimile No. (703) 305-3230 



Aulh 



LEOBOUDREAU 



Telephone No. (703) 308-7595 



Form PCT/ISA7210 (second sheet)(Juiy 1992)* 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US96/I0164 



A. CLASSIFICATION OF SUBJECT MATTER: 
US CL : 

382/103, 104, 105, 107, 173, 181, 190, 209, 217, 218, 219, 220, 221, 232, 236, 238, 251. 252, 253 



Form PCT/ISA/2I0 (extra sheet)(July 1992)* 



